High dynamic range imaging

ABSTRACT

A high dynamic range video processing method performs merging and tone mapping techniques after a Bayer filter mosaic technique is performed and then converts it to red green blue (RGB) at the end as opposed to converting into RGB at the beginning and then performing merging and tone mapping after. The HDR processing is performed on Bayer-mosaic images and no de-mosaicing and color space conversions are required. The merging procedure has two modes: full-reset merging and LDR-updated merging. The first mode, full-reset merging, creates an HDR frame once the system has all image frames captured. The second mode, LDR-updating merging, means that any new HDR frame is obtained by an updating of a previous HDR frame with a new LDR frame data.

This application is a Continuation of co-pending application Ser. No.15/272,904, filed on Sep. 22, 2016, currently pending, for whichpriority is claimed under 35 U.S.C. § 120 and the entire contents of allof which are hereby incorporated by reference.

BACKGROUND OF THE INVENTION Field of the Invention

The present invention relates to digital imaging. More specifically, thepresent invention discloses a method and device for high dynamic rangevideo processing that processes a smaller stream of data to achieve ahigh frame rate.

Description of the Prior Art

High dynamic range imaging is used to reproduce a greater dynamic rangeof luminosity in imaging and photography.

A conventional technique of high dynamic range imaging includesutilizing special image sensors for oversampling. Another techniqueinvolves merging multiple images.

However, the special image sensors often encounter difficulty when usedin low light conditions which produces a non-optimal resultant image.

Additionally, digital image encoding does not always offer a greatenough range of values to allow fine transitions which causesundesirable effects due to lossy compression.

Therefore, there is need for an efficient method and device for highdynamic range video processing that produces superior high dynamic rangevideo images at a high frame rate.

SUMMARY OF THE INVENTION

To achieve these and other advantages and in order to overcome thedisadvantages of the conventional method in accordance with the purposeof the invention as embodied and broadly described herein, the presentinvention provides an efficient method and device for ultra high dynamicrange video processing that produces superior high dynamic range videoimages at a high frame rate.

The present invention provides a hardware realization of improved ultrahigh dynamic range (HDR) technology. The present invention processes,merges, and tone maps multiple exposures in a video form using a fieldprogrammable gate array (FPGA) platform.

The method provides a unique way to perform the merge and tone mappingtechniques after a Bayer filter mosaic technique is performed and thenconvert it to red green blue (RGB) at the end as opposed to convertinginto RGB at the beginning and then performing merging and tone mappingafter. In this way the present invention has a significantly smallerstream of data being processed which allows for achieving higher framerates.

The linear primary Bayer mosaic signals are converted directly to alogarithmic scale pixel-wise. Some color balance pre-compensation can beused before the conversion. In this way, each R, G1, G2, or B pixel isbeing converted to its logarithmic value independently. Then the pixelsare processed and the HDR result is converted from the log-scale back tothe primary linear Bayer mosaic. This helps to insert the processingbetween an image sensor and a commonly used image processor.

The ultra high dynamic range imaging of the present invention allows athroughout compatibility for all 3 main stages: merging, tone mapping,and compression.

Additionally, the present invention preserves both the details andcolors of the captured HDR scene.

Furthermore, in the present invention, the resulting images look asnatural as possible within the capabilities of the capturing andreproduction devices.

The HDR processing of the present invention is performed on Bayer-mosaicimages (RAW data from an image sensor). No de-mosaicing and color spaceconversions are required to perform merging and tone mapping operations.This allows for saving processing resources and decreasing color losses.

All HDR processing operations are performed in a logarithmic scale tomeet human eye vision aspects. This method significantly simplifiescalculations.

For merging operations, N image frames (of different exposures) are usedper HDR capture.

The merging procedure has two modes: full-reset merging and LDR-updatedmerging. The first mode, full-reset merging, creates an HDR frame oncethe system has all N frames captured. The second mode, LDR-updatingmerging, means that any new HDR frame is obtained by an updating of aprevious HDR frame with a new LDR (low dynamic range) frame data. Thus,the HDR frames are updated by LDR (low dynamic range) frames at a framerate of LDR frames.

For example: LDR frames come at 120 fps, then the first mode gives 30fps for HDR images, the second mode gives 120 fps for HDR images.

For some FPGA designs, a 16-bit operation limits the HDR range to 16 EV(exposure value). But even this allows for covering all the exposurerange settings of an image sensor and the exposure time can becontrolled via a timing of the sensor only.

Additionally, the output HDR image is a Bayer-mosaiced HDR image.

Locally-adaptive tone mapping performs a brightness range compression ina human-eye comfortable manner. The tone mapping is human-eye oriented.In other words, the present invention tone maps the images with the useof an artist painting approach.

Color chromaticity is preserved during the tone mapping process. Colordistortions are minimal, depending on the Bayer mosaic type and opticalproperties of the lens sensor system. This is provided by the ability totone map Bayer-mosaiced images in primary sensor colors (without acolor-space conversion and de-mosaicing).

For example, the present invention can compress the HDR brightness rangefrom 96 dB to 8-bit per pixel output and the output HDR image is aBayer-mosaiced tone mapped HDR image.

When using 32-bit processing, the merging can give up to 32 EV HDRimages depending on the image sensor.

The tone mapping can compress from 192 dB, with the use of locallyadaptive calculations, to 8-bit images.

These and other objectives of the present invention will become obviousto those of ordinary skill in the art after reading the followingdetailed description of preferred embodiments.

It is to be understood that both the foregoing general description andthe following detailed description are exemplary, and are intended toprovide further explanation of the invention as claimed.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings are included to provide a furtherunderstanding of the invention, and are incorporated in and constitute apart of this specification. The drawings illustrate embodiments of theinvention and, together with the description, serve to explain theprinciples of the invention. In the drawings:

FIG. 1A is a flowchart illustrating a method for high dynamic resolutionvideo processing according to an embodiment of the present invention;

FIG. 1B is a flowchart illustrating merging techniques for high dynamicresolution video processing according to an embodiment of the presentinvention;

FIG. 1C is a flowchart illustrating a method for high dynamic resolutionvideo processing according to an embodiment of the present invention;

FIG. 2 is a drawing illustrating a device for high dynamic resolutionvideo processing according to an embodiment of the present invention;

FIG. 3 is a drawing illustrating the additive gamma-like exposurecorrection function Δb(b_(P));

FIG. 4 is a drawing illustrating an elementary detail through asingle-step transition b₁b₁→b₂ within a circle area C of radius r with acenter o at a point of interest (POI);

FIG. 5 is a drawing illustrating the b₁-b₂ transition along across-section line l drawn in FIG. 4;

FIG. 6 is a drawing illustrating the single-step transition of the localLEC change (near r′≅0); and

FIG. 7 is a drawing illustrating a 5×5 kernel example of a 2D-kerneldistribution of RGGB pixels.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

Reference will now be made in detail to the preferred embodiments of thepresent invention, examples of which are illustrated in the accompanyingdrawings. Wherever possible, the same reference numbers are used in thedrawings and the description to refer to the same or like parts.

Refer to FIG. 1A, which is a flowchart illustrating a method for highdynamic resolution video processing and to FIG. 1B, which is a flowchartillustrating merging techniques for high dynamic resolution videoprocessing.

As shown in FIG. 1A the high dynamic resolution video processing method100 of the present invention begins in Step 110 by receiving incominglight. The light passes through a Bayer filter in Step 120 and iscaptured by an image capture device such as, for example, a sensor arrayin Step 130.

The RAW data from the image capture device is provided to a HDRprocessor module for processing. In Step 140 merging techniques areperformed on the RAW data and in Step 160 tone mapping techniques areperformed.

In Step 180 the merged-tone mapped data is converted to RGB and the HDRimage is output in Step 190.

The HDR video processing method 100 of the present invention provides aunique way to perform the merge and tone mapping techniques after aBayer filter mosaic technique is performed and then convert it to redgreen blue (RGB) at the end as opposed to converting into RGB at thebeginning and then performing merging and tone mapping after. In thisway the present invention has a significantly smaller stream of databeing processed which allows for achieving higher frame rates.

The HDR processing of the present invention is performed on Bayer-mosaicimages (RAW data from an image sensor). No de-mosaicing and color spaceconversions are required to perform merging and tone mapping operations.This allows for saving processing resources and decreasing color losses.

All HDR processing operations are performed in a logarithmic scale tomeet human eye vision aspects. This method significantly simplifiescalculations.

For merging operations, N image frames (of different exposures) are usedper HDR capture. The method supports any amount of frames of differentexposures to be merged.

The merging procedure (Step 140) has two modes: full-reset merging andLDR-updated merging. The first mode (Steps 145, 150 FIG. 1B), full-resetmerging, creates an HDR frame once the system has all image framescaptured. The second mode (Step 155 FIG. 1B) LDR-updating merging, meansthat any new HDR frame is obtained by an updating of a previous HDRframe with a new LDR (low dynamic range) frame data. Thus, the HDRframes are updated by LDR (low dynamic range) frames at a frame rate ofLDR frames.

For example: LDR frames come at 120 fps, then the first mode gives 30fps for HDR images, the second mode gives 120 fps for HDR images.

For some FPGA designs, a 16-bit operation limits the HDR range to 16 EV(exposure value). But even this allows for covering all the exposurerange settings of an image sensor and the exposure time can becontrolled via a timing of the sensor only.

Additionally, the output HDR image is a Bayer-mosaiced HDR image.

Locally-adaptive tone mapping performs a brightness range compression ina human-eye comfortable manner. The tone mapping is human-eye oriented.In other words, the present invention tone maps the images with the useof an artist painting approach.

Color chromaticity is preserved during the tone mapping process. Colordistortions are minimal, depending on the Bayer mosaic type and opticalproperties of the lens sensor system. This is provided by the ability totone map Bayer-mosaiced images in primary sensor colors (without acolor-space conversion and de-mosaicing).

For example, the present invention can compress the HDR brightness rangefrom 96 dB to 8-bit per pixel output and the output HDR image is aBayer-mosaiced tone mapped HDR image.

When using 32-bit processing, the merging can give up to 32 EV HDRimages depending on the image sensor.

The tone mapping can compress from 192 dB, with the use of additionaledge-processing calculations, to 8-bit images.

Refer to FIG. 1C, which is a flowchart illustrating a method for highdynamic resolution video processing according to an embodiment of thepresent invention.

The high dynamic resolution processing method 200 comprises convertinglinear primary Bayer mosaic signals directly to a logarithmic scalepixel-wise in Step 210. In this way, each R, G1, G2, or B pixel isconverted to its logarithmic value independently. In Step 220, thepixels are processed to obtain a high dynamic range result. And in Step230, the high dynamic range result is converted from the logarithmicscale back to the primary linear Bayer mosaic.

Refer to FIG. 2, which is a drawing illustrating a device for highdynamic resolution video processing according to an embodiment of thepresent invention.

The high dynamic resolution video processing device 10 of the presentinvention comprises a Bayer filter 50, an image capture device 60, andan HDR processor or HDR processing module 80.

In an embodiment of the present invention the Bayer filter 50 and theimage capture device 60 are separate devices from the HDR processor 80.In another embodiment of the present invention the image capture device60 is integrated with the HDR processor 80.

In an embodiment of the present invention the HDR processor 80 comprisesa field programmable gate array (FPGA).

In operation, light 40 is received by the HDR video processing device10. The light 40 passes through a Bayer filter 50 and is captured by animage capture device 60 such as, for example, a sensor array.

The RAW data 70 from the image capture device 60 is provided to the HDRprocessor module 80 for processing. The HDR processor module 80 performsmerging techniques on the RAW data and tone mapping techniques.

The merged-tone mapped data is then converted to RGB and the HDR image190 is output.

Basic requirements for high dynamic range imaging (HDRI) are targeted toachieve human eye abilities in terms of the dynamic range: once a humaneye observes all the highlights and shadows of the perceptible scene,then the HDRI system should be able to save and reproduce the samevisual data. This means, that the HDRI system should work at an absoluteexposure range of 42 . . . 46 EV stops (human eye absolute range), or atleast be locally adapted at the exposure range of 22 . . . 24 EV stops(human eye common viewable range).

In most cases, dynamic ranges of image sensors are not sufficient,varying from 8 EV stops (for low-cost cameras) to ˜14 EV stops (forhi-end cameras). In order to extend a dynamic range of images, anexposure bracketing is applied; a set of LDR images (brackets) takenwith different exposure settings are being produced for the same scene.Brackets can include different settings for exposure time, gain (or ISOspeed) and aperture. Then, a merging procedure is applied to the set ofLDR images in order to obtain a final HDR image. The merging qualitydepends on the bracketing method used.

In exposure time bracketing, different LDR images are taken withdifferent exposure time settings. A whole image should be readout froman image sensor for each bracket. This method gives the biggest possibledynamic range for the given sensor usage, because all the range ofexposure time setting can be used.

In gain (or ISO speed) bracketing, different LDR images are taken withdifferent Gain settings. In such a method the image is being taken justonce, and then it is being kept in an analog buffer of the image sensor.Brackets are formed by a multi-readout process of the image from theanalog buffer using different gains for different frames readout. Themerging procedure does not require a de-ghosting correction since thereare no motions between the frames readouts.

Aperture bracketing requires a mechanical aperture operation. Theaperture is being changed for each bracket, thus defining a light fluxrange to be captured within the given exposure time. This kind ofbracketing can be applied at a constant exposure time and a constantgain, so the brackets (contrary to other brackets types) have the sameSNR (signal-to-noise ratio). This makes the exposure time as big aspossible and helps to achieve the best possible SNR for the givensensor.

A combination of bracketing types is used to achieve better SNR or fewerof ghost artifacts.

For HDR images of 24 EV range the data size is 4 times bigger, than thatfor “regular” 8-bit LDR images. Good compression techniques withminimized color losses are utilized to save the HDR images or transmitthem via communication protocols.

Widely used reproduction devices, such as monitors/displays or printers,do not have more than 8 . . . 12 EV stops of DR. In order to reproduceHDR images (with a DR higher than 8 . . . 12 EV) on these devices, atone mapping (TM) procedure is performed in order to map all brightnessvalues (of all pixels in the image) into a target DR of the device. Thisis also known as dynamic range compression.

Tone mapping is a complicated and sophisticated procedure. With tonemapping all highlights and shadows of an HDR image should be seen on theLDR reproduction device (global contrast modification), the visualcontrast of the image's details should not be lost (local contrastmodification), and the colors of the details (chromaticity) should notbe changed (or color losses should be minimized).

The following details some challenges with tone mapping techniques.

-   -   Any image (LDR or HDR) is represented through its pixels,        denoted as l=l(x,y), where:    -   Symbol l represents one of the pixels color components l=(q₀, .        . . , q_(n)) in the given color domain (for example, l=(R,G,B),        or l={L,u,v})    -   Values (x,y) represent spatial coordinates of the pixel in the        image.

There are different tone-mapping techniques, which can be divided intosimple “pixel-wise” TM and TM with details “preservation”.

For simple “pixel-wise” TM, a predefined tone-mapping curve T(l) is usedfor any pixel (the curve is common for all pixels). In this case noimage details (bigger than 1×1 pixel) are taken into account; a valuel=l(x,y) of a pixel is being mapped through the curve to get a new valuel_(TM)=l_(TM)(x,y) for the pixel in the final tone-mapped image as:

l _(TM) =T(l)

An advantage of simple “pixel-wise” tone mapping is that theimplementation is rather simple and it allows embedding the TM-curvefunctionality into an image sensor as “on-chip” solution. However,contrast losses occur for details, when the HDR image is of DR>10 EV,because the transfer function (which is the same for all pixels in theimage) “knows nothing” about the image details: the predefined TM curvewill not match some detail's brightness range.

In addition to the TM-curve mentioned above, in TM with details“preservation” spatial frequencies are being analyzed in an HDR image topreserve the image details. In this case the image is divided into twolayers: High-frequency layer l_(HF)=l_(HF)(x,y) for details andLow-frequency layer l_(LF)=l_(L,F)(x,y)for other data. This pixel-wiselayers separation looks like:

l=l _(HF) +l _(LF)  (2.1)

The l_(HF) is obtained via high-pass filtering of the image l (using,for example, a Fourier transform), while the l_(LF) is calculated from(1) by using the image l and the l_(HF) layer as:

l _(LF) =l−l _(HF)

The approach (1) can be interpreted as an equivalent separation ofincident light energy in terms of spatial distribution of the energyover the sensor's pixels.

Global Contrast: the l_(LF) is being modified (usually, by a Gamma-likeTM-function) to map the required HDR range into the target monitorrange:

l _(LF) ^(TM) =T(l _(LF))  (2.1.1)

Local Contrast: the dT(l_(LF))/dl_(LF) is being additionally modified atcertain ranges of l_(LF). So, for these ranges the contrast can be madelower or higher.

Details contrast, or micro-contrast: the l_(HF) is being modified interms of its amplitude; usually it is modified by some amplitude factorA:

l _(HF) ^(TM) =A*l _(HF)  (2.1.2)

Then, the operation (1) is used to get the tone-mapped image:

l ^(TM) =l _(HF) ^(TM) +l _(LF) ^(TM)  (2.2)

Known results of the calculations (2) produce “unnaturally looking”images, because the contrast operations above are being performed in thelinear energy-to-digit scale, but the human eye perceives the contrastin a logarithmic manner. In addition, the Fourier transform can be toocalculative, for example, for video applications. That's why, in orderto make the tone mapping “more natural” and to increase the performanceof the HDRI system, the tone-mapping uses a logarithmic representationof the layers l,l_(HF),l_(LF).

B=log_(l)l, B_(HF)=log_(l)l_(HF), b_(LF)=log_(l)l_(LF),

Where l is a logarithm base, which is usually equal to 2 (known as EVscale), so 2×-change of the l relates to 1 EV “stop”.

In this case, the pixel-wise layers separation is more similar to thehuman eye vision system:

B=B _(HF) +B _(LF)  (2.3)

In accordance with the human eye operation, the B_(LF) is beingcalculated from B by an application of a low-pass filtering L(B|r) witha Gaussian 2D-filter (kernel) of a given standard deviation r:

B _(LF) =L(B|r)

The parameter r is also known here as an “effective radius” of thefilter.

Then the B_(HF) is calculated from (3) as:

B _(HF) =B−B _(LF)

The pixel-wise tone mapping operation here is similar to (2), but in alogarithmic scale:

B ^(TM) =A*B _(HF) +T(B _(LF))  (2.4)

Layers B_(HF) and B_(LF) are also known as local contrast layer B^(L)and global contrast layers B^(G) accordingly

B^(L)=B_(HF) B^(G)=B_(LF)  (2.5)

This approach is closer to human eye vision properties, thus it canpotentially produce better results in a perception of the HDRtone-mapped image. However, the operation (2.4) just approximates thehuman vision system; it actually works for small details of 4 degreesviewing angle, so, when the approach (2.4) is used, the image qualitydepends on the distance of the image observation (a distance from ahuman eye to a displaying device, where the image is reproduced); thus,the distance of the image observation depends on the given parameter rof the Gaussian filtering. Also, the details contrast cannot beefficiently adjusted for different spatial frequencies, since a single ris used; bigger r values lead to noticeable “halo” for high-contrastedges in the image. Additionally, the A factor should not be constant,because, for example, “halos” can be unnoticeable for a group ofdetails, but can appear for a single detail on a homogeneous brightfield of the image.

Ideally, an electrical signal from a pixel can be represented as asignal proportional to some amount of incident light energy with aspectral distribution s(λ) coming through an optical aperture f anddetected within the pixel's area of an effective spectral sensitivityp(λ) during a time period τ (“exposure time”), so the pixel's measuredvalue is:

$I = {A\frac{\alpha\tau}{f^{2}}{\int_{0}^{\infty}{{p(\lambda)}{s(\lambda)}d\; \lambda}}}$

where A is a proportionality constant; α is any amplification gain (i.e.analog, digital).

Note:

P _(s) =A∫ ₀ ²⁸ p(λ)s(λ)dλ

as a color component value (depends on the pixel's effective spectralsensitivity p(λ)/“primary color filter”/and an incident light spectrums(λ)), and:

$ɛ = \frac{\alpha\tau}{f^{2}}$

as a camera exposure factor (depends on the camera exposure settings).

Then,

l_(P)=εP_(s)  (3.1)

In most cases, only ε exposure factor can be controlled through camerasettings, while P_(s) can partially depend on uncontrolled dynamicallychanged illumination conditions of a captured scene: for example,environment light conditions along with artificial lights (likeflashlights).

An exposure change factor is introduced:

β=β₀β_(e)β_(s)  (3.2)

where:

β_(e)=e_(o)/e_(s) is a camera exposure change factor,

β_(g)=P_(s2)/P_(s1) is an exposure change factor related to a sceneillumination change,

β_(c)—intended for a post-processing exposure correction.

Each β value can be defined through a bracketing sequence.

From (3.1) and (3.2), the exposure change produces a change in thepixel's value:

l_(P) ^(new)=βl_(P)  (3.3)

As a particular case, the exposure change can be performed at a certainpixel within an image sensor.

The equation (3.3) is true for an ideal sensor. But any transferfunction of any image sensor is nonlinear. Even if the transferfunction's working range is linear as (1), it has at least twosaturation levels:

l=l_(P) ^(max) refers to a highest value (camera saturation level) forthe given sensor's (camera) transfer function. For example, for 8 bitsper pixel (LDR) l_(P) ^(max)=255; for N bits per pixel (N>8, HDR) l_(P)^(max)=2^(N)−1.

l=0 refers to a lowest saturation level for the given sensor's transferfunction, when the incident light energy is below a minimal detectablevalue for the sensor.

In the present invention a logarithmic scale is used for the pixels'values representation:

B_(P)=log_(l)l_(P)

Saturation levels of a transfer function will limit the B_(P) range too,so the range of the pixel's values in B representation (B-scale) is:

B∈[−∞,B_(P) ^(max)], where B_(P) ^(max)=log_(l)l_(P) ^(max).

The B values are renormalized into “decibel”-like values, where themaximal value (or “white” point) is equal to zero:

b _(P) =B _(P) −B _(P) ^(max)  (3.4)

where:

b_(P)∈[−∞,0]

The B_(P) ^(max) depends on the camera system used, but it can also beset to any convenient reference value for a normalization required.

In order to display the pixels' values after a processing in alogarithmic b-scale, the following backward conversion is used:

l=l ^(b) ^(P) ^(+B) ^(M) ^(max),

-   -   where B_(M) ^(max) is a dynamic range of a displaying monitor in        the B-scale.

The exposure change (3.2) produces a change in the pixel's visualbrightness on a reproduction device, such as a monitor.

In a logarithmic B-scale, the exposure change made by a camera and/orscene illumination can be expressed through the following equation:

B _(P) ^(new)=log_(l) l _(P) ^(new)=log_(l)(βl _(P))=log_(l) l_(P)+log_(l) β=B _(P) +ΔB _(P)

where the exposure change is:

ΔB_(P)=log_(l)β  (3.5)

So, here the ΔB_(P) represents an additive exposure change (and then—avisual brightness change) via β parameter.

From equations (3.2) and (3.5), the total exposure change and correctioncan be expressed through a sum of its additive components:

ΔB _(P)=log_(l)β_(c)+log_(l)β₂+log_(l)β_(S)

so it is denoted as:

ΔB _(P) =ΔB _(P) ^(c) +ΔB _(P) ^(e) +ΔB _(P) ^(s)  (3.6)

where:

ΔB_(P) ^(o)=log_(l)β_(c) ΔB_(P) ^(e)=log_(l)β_(e) ΔB_(P)^(s)=log_(l)β_(s)

Thus, any modification/correction of pixel brightness can be performedas an exposure change ΔB, where the ΔB value (3.5) can be obtained fromany of the basic components (3.6):

ΔB_(P) ^(c)—illumination change (i.e. using flashlights),

ΔB_(P) ^(e)—camera exposure change (time, gain, aperture),

ΔB_(P) ^(o)—mathematical “exposure correction”.

For example, if the exposure changes ΔB_(P) ^(s) and ΔB_(P) ^(e) poduceinsufficient visual brightness of a pixel on a reproduction device, thensome additional correction of the pixel's exposure can be performed“mathematically” using ΔB_(P) ^(c).

Since any ΔB value is additive, the following equations are also truefor the b-scale:

b _(new) =b _(P) +Δb _(P)  (3.7)

Δb _(P) =Δb _(P) ^(c) +Δb _(P) ^(e) +Δb _(P) ^(s)

Δb_(P) ^(s)=ΔB_(P) ^(s), Δb_(P) ^(e)=ΔB_(P) ^(e), Δb_(P) ^(c)=ΔB_(P)^(c)

-   -   From here, a pixel's value modification is defined in terms of        “exposure values”: b_(P) is an exposure value of a pixel, and        Δb_(P) is an exposure change of the pixel.

In order to transform pixels' brightness values from one range toanother (i.e. HDR-to-LDR range transform—making all pixels from HDRimage viewable on an LDR reproduction device), a tone mapping operationis used. From its definition, the tone mapping actually performs anexposure range compression or expansion.

The tone mapping operation is defined as a mapping of b_(P)-values froma source b-range to a target b-range. The mapping operation is beingperformed through an additive function:

Δb=Δb(x,y,b)

which performs an additive exposure correction of a pixel at(x,y)-coordinates in the image as:

b _(new) =b _(P) +Δb(x,y,b _(P))  (3.8)

b _(P) =b _(P)(x,y)

The function is represented in the same components as (3.7):

Δb _(P)(x,y,b _(P))=Δb _(P) ^(s)(x,y,b _(P))+Δb _(P) ^(e)(x,y,b _(P))+Δb_(P) ^(c)(x,y,b _(P))

To build a “naturally working” tone mapping function, the tone mappingshould work similarly to a human eye local adaptation; it should help tovisualize dark areas of the image making them brighter, while keepingthe brightest areas observable. This human eye ability (localadaptation) is well approximated by a Gamma function in a linear (lightenergy) scale. When a low dynamic range reproduction device is used todisplay an image, then an additional gamma correction is applied to allpixels in the image:

${G_{P}( I_{P} )} = {A*{{I_{P}^{\max}( \frac{{\alpha I}_{P}}{I_{P}^{\max}} )}^{\frac{1}{\gamma}}.}}$

Here G_(P)(I_(P)) performs exposure range compression in terms ofexposure compensation, which “helps” the human eye to observe imagepixels, whose values l_(P) are out of a dynamic range of thereproduction device. The γ-correction parameter allows observing darkerareas of the image, while keeping the brightest elements. The αparameter performs a rescaling/amplification of an input pixel's value(or exposure). The A parameter can be considered as an output luminancerescaling/amplification factor for a reproduction device.

In a logarithmic B-scale, the gamma correction is simply a linearfunction:

$B_{G} = {{{\log \;}_{l}{G_{P}( I_{P} )}} = {{\log_{l}\lbrack {A*{I_{P}^{\max}( \frac{{\alpha I}_{P}}{I_{P}^{\max}} )}^{\frac{1}{\gamma}}} \rbrack} = {{\Delta \; b_{out}} + B_{P}^{\max} + {\frac{1}{\gamma}( {B_{P} + {\Delta \; b_{in}} - B_{P}^{\max}} )}}}}$  where:   Δ b_(out) = log_(l) A, Δ b_(in) = log  _(l)a.

When normalized to a b-scale (3.4), the equation above is even simpler:

$b_{G} = {{\frac{1}{\gamma}( {b_{P} + {\Delta \; b_{in}}} )} + {\Delta \; b_{out}}}$

As it can be seen from the last two equations, the gamma-functionperforms a linear compression/expansion in the logarithmic b-scale (orB-scale) by means of a compression factor 1/γ. The function alsoperforms an input additive exposure correction of a pixel by Δb_(HF)value and an output pixel exposure (luminance) correction by means ofΔb_(out) value.

This operation can be expressed through an additive exposure correction(3.8). Note:

$\rho:={1 - \frac{1}{\gamma}}$b_(G) = b_(P) + Δ b_(in) − ρ(b_(P) + Δ b_(in)) + Δ b_(out)

so, the basic gamma-like additive exposure correction for any pixel is:

Δb(b _(P))=Δb _(in) −P(b _(P) +Δb _(in))+Δb _(out)  (3.9)

where:

-   -   Δb_(in) performs a preliminary exposure correction of a pixel,    -   ρ defines a dynamic range compression, it also works as a global        contrast factor,    -   Δb_(out) performs a final exposure correction of a pixel, and        also works as an output luminance change.

Refer to FIG. 3, which is a graph illustrating the additive gamma-likeexposure correction function Δb(b_(P)).

From the definition l_(LF) ^(TM)=T(l_(LF)) (2.1.1), the ρ parameter ofΔb(b_(P)) function modifies a global contrast of the image. It applies acompression or extension of a total input range of pixel's exposures. Onthe other hand, it modifies exposure values similar to a localadaptation of a human eye.

In an embodiment of the present invention, the function (3.9) behavior(over its parameters) is considered as a local exposure compensation(LEC), which is used to compensate a local adaptation of a human eye,when an image is being viewed at a reproduction device.

Δb(b _(P))=Δb _(in) −P(b _(P) +Δb _(in) −b ₀)+Δb _(out)  (3.10)

Thus, when a maximal input exposure value maxb_(P)=−Δb_(in) is set, thenthe ρ, b₀ and Δb_(out) parameters help to compensate local exposures fora better observation of dark areas of the image at a reproductiondevice. The compensation is performed in terms of human eye localadaptation.

Further, Δb_(in) and Δb_(out) are the same constants for all b_(P)(x,y),and Δb_(in) is set to have a normalization b_(P,max)=0. In this case,the function (3.1) can look like

Δb(b _(P))=−ρb _(P) +Δb _(out)  (3.11)

While the function (3.10) has an advantage in performing of localexposure compensation, its disadvantage is obvious from theGamma-function definition: when adjusted to compress high-contrastscenes, it also compresses a native contrast of brighter details of theimage, making them visually unnatural. On the other hand, if anycontrast compression is prohibited or masked, the LEC does not makesense—nothing will be changed in the image after the “compensation”.

Thus, the tone mapping function cannot consist of the function (3.10)only. A good balance between local exposure compensation and sufficientvisual contrast should be kept. That's why the LEC function (exposurecompensation) should be locally adaptive to a native contrast of theimage details.

In an embodiment of the present the LEC local adaptation is consideredas a reaction on a native micro-contrast of a detail.

Consider an elementary “detail” (known as “edge”) through a single-steptransition b₁b₁→b₂ ithin a circle area C of radius r with a center o ata point of interest (POI), as shown in FIGS. 4 and 5.

FIG. 5 shows the b₁→b₂ transition along a cross-section line l drawn inFIG. 4. The line l is perpendicular to the transition edge e. The vectorr′ is in the direction of l. In an embodiment the native micro-contrastis defined through a local exposure deviation of the value b_(P) at thepoint o from an average value b _(r) calculated over all b-values withinthe C area as:

D _(r)(b _(P))=b _(P) −b _(P)

For example, for the point o shown in FIG. 4 b_(P)=b₁ is obtained. Themaximal value of D_(r)(b_(P)) can be found near r¹≅0.

Since the point o has coordinates (x,y),

b _(P) =b _(P)(x,y)

The LEC function (3.10) is made locally adaptive through a localmodification of the compression factor ρ. In order to keep or amplify amicro-contrast D_(r)(b_(P)) of a detail, the slope of the LEC function(defined by ρ=ρ_(o) parameter) should be locally decreasedρ=ρ_(local)ρ_(o) within the C area, if any local exposure deviationD_(r)(b_(P)) in the area is nonzero.

For the single-step transition the proposed local LEC change (near r′≅0)is represented in FIG. 6.

In accordance with FIG. 6 and equation (3.11), the locally modified LECbecomes dependent on the r and then can be written as follows:

Δb _(P)(b _(P))=−ρ_(local) D _(r)(b _(P))−ρ₀ b _(r) +Δb _(out)  (3.12)

where

b _(P)(b _(P)(x,y) b _(r) =b _(T)(x,y)

The function (3.12) has the following properties over the ρ_(local)parameter:

-   -   when ρ_(local)=0, a micro-contrast preservation results. This is        the basic case—no change with the micro-contrast happens within        the C area. Thus, when the exposure correction Δb_(P)(b_(P)) is        added to initial image b_(P):

b _(out) =b _(P) +Δb _(T)(b _(P))=b _(P)−ρ₀ b _(r) +Δb _(out)

It can be written: b_(P)=(b_(p)−b _(r))+b _(r), so

b _(out)=(b _(P) −b _(T))+(1−ρ₀)b _(r) +Δb _(out) =D _(T)(b _(P))+(1−ρ₀)b _(P) +Δb _(out)

As can be seen, ρ₀ does not change the micro-contrast D_(r)(b_(P)) inthe logarithmic scale.

For other ρ_(local):

ρ_(local)>ρ₀: micro-contrast suppression;

ρ_(local)=ρ₀: the equation (3.12) turns into (3.11)—no micro-contrastamplification; just gamma-compression.

ρ_(local)<ρ₀: micro-contrast amplification;

Parameter ρ₀ works as a global gamma-compression parameter.

The following details micro-contrast to local contrast.

If a Gaussian-weighted averaging (with dispersion r²) is used tocalculate b _(r)

$\begin{matrix}{{{\overset{\_}{b}}_{r}( {x,y} )} = {{\frac{1}{2\; \pi \; r^{2}}{\int_{- \infty}^{+ \infty}{\int_{- \infty}^{+ \infty}{{b_{P}( {{x - x^{\prime}},{y - y^{\prime}}} )}e^{- \frac{x^{\prime 2} + y^{\prime 2}}{2r^{2}}}{dx}^{\prime}{{dy}^{\prime}.{{\overset{\_}{b}}_{r = 0}( {x,y} )}}}}}}:={b_{p}( {x,y} )}}} & (3.13)\end{matrix}$

The equation (3.12) becomes working in the same way as the approachdescribed in regard to equations 2.3, 2.4, 2.5, relating to a humanvision system. Then, the image b_(P)(x,y) is being separated into twolayers: G (“global contrast” part) and L (“local contrast” part):

b _(P)(x,y)=b _(T) ^(L)(x,y)+b _(T) ^(G)(x,y)

b _(T) ^(G)(x,y)= b _(r))(x,y)

-   -   and b _(P)(x,y) is calculated from (3.13).

Here, details of characteristic sizes about or less than the value r(spatial frequencies higher than ˜1/r) are in the layer b_(P) ^(L)(x,y),which incorporates the “local contrast” data (analogous to the“micro-contrast” D_(T)(b_(P)) described above)

b _(T) ^(L)(x,y)=b _(P)(x,y)−b _(T) ^(G)(x,y)

Then, the equation (3.12) can be rewritten in the following way

Δb _(T)(b _(P))=−ρ_(local)b_(T) ^(L)−ρ_(global) b _(T) ^(G) +Δb_(out)  (3.14)

where ρ_(global) operates the global contrast correction (same asρ_(o)), while ρ_(local) operates the local contrast of the imagedetails. The equation (3.14) has the same properties as (3.12):

ρ_(local)=0: local contrast preservation;

ρ_(local)>ρ_(global): local contrast suppression;

ρ_(local)=ρ_(global): no local contrast amplification, justgamma-compression;

ρ_(local)<ρ_(global): local contrast amplification.

The equation (3.14) may still have the same disadvantages of theapproach described regarding equations 2.3, 2.4, 2.5, including “halo”artifacts.

In order to resolve insufficient local contrast in the Gamma compressionoperation and eliminate the “halo” artifacts problems, an approach isutilized where HDR images are processed not only by human eye relatedcalculations (3.14), but also for human eye natural-like perception,“like artists create paintings”. To achieve this Locally Adaptive ToneMapping (LATM) is utilized, where:

1. Any local contrast modification being performed at a point (x,y)should work as an additive contrast compensation applied to the nativelocal contrast deviation b_(P) ^(L) found at this point in the originalimage b_(P).

2. Resulting local contrast deviations should be as close as possible tothe original deviations b_(P) ^(L), while all b_(P) ^(L) values arestill mapped into a limited available contrast range of a reproductiondevice (or reflection density of paints, inks and so on).

3. Resulting local contrast deviations should be visually invariant toany distance of their observation, thus the additive local contrastcompensation should be processed for each available spatial frequency(−1/r) independently.

From here, the ρ_(local) factor should be locally adaptive to nativelocal contrast (NLC) deviations b_(r) ^(L) at each point of (x,y) in arelation to r

ρ_(local)=ρ_(local)(x,y,r)

Solution for statement 1 above. Additive local contrast compensation isalready expressed in the equation (3.14) as a “portion” of an originaldeviation b_(P) ^(L) as: −ρ_(local)b_(r) ^(L).

For the sake of operational convenience, the local contrast modificationparameter ρ_(local) is rewritten as:

ρ_(local)=ρ_(L)k_(P) ^(L)

so,

ρ_(local)b_(T) ^(L)=ρ_(L)k_(T) ^(L)b_(T) ^(L)

where k_(P) ^(L) is being modulated at a point (x,y) by an originaldeviation b_(P) ^(L) this point as

k _(P) ^(L) =k _(L)(x,y,r)=k ^(L)(b _(P) ^(L)(x,y))

and the ρ_(L) parameter is a common scaling factor, which does notdepend on b_(T) ^(L) and (x,y) coordinates.

The additive local contrast compensation is defined from the followingequation

b _(r,paint) ^(L) =b _(r) ^(L) −k _(T) ^(L) b _(T) ^(L)

0<k_(T) ^(L)≦1

where b_(r,paint) ^(L) denotes a resulting local contrast deviation,which is supposed to be “painted” at a reproduction device.

Solution for statement 2 above. To define an adaptive behavior of k_(r)^(L) an approximation model is used, which is described by the following“balanced contrast” relationship

$\frac{b_{r,{paint}}^{L}}{b_{\max}^{L}} = \frac{b_{r}^{L} - b_{r,{paint}}^{L}}{b_{r}^{L}}$b_(max)^(L) > 0 0 ≤ b_(t, paint)^(L) ≤ b_(max)^(L)

Here, b_(max) ^(L) value is a maximal available contrast range at areproduction device.

From statements 1 and 2 above:

$\begin{matrix}{b_{r,{paint}}^{L} = {\frac{b_{r}^{L}}{1 + \frac{b_{r}^{L}}{b_{\max}^{L}}}.{Then}}} & (3.15) \\{k_{r}^{L} = {\frac{h_{s}{b_{r}^{L}}}{1 + {h_{s}{b_{r}^{L}}}}.}} & (3.16)\end{matrix}$

where h_(s) can be considered a halo suppression coefficient

$h_{s} = \frac{1}{b_{\max}^{L}}$

The halo suppression coefficient h_(s) is a positive h_(s)≧0 constant,which does not depend on (x,y,r) and can be operated by a user.

Equation (3.16) with statement 1 defines a locally adapted localcontrast layer

$\begin{matrix}{b_{r}^{LA} = {{k_{r}^{L}b_{r}^{L}} = {\frac{h_{s}{b_{r}^{L}}}{1 + {h_{s}{b_{r}^{L}}}}b_{r}^{L}}}} & (3.17)\end{matrix}$

Using the b_(r) ^(LA) layer from (3.17) in equation (3.14), the globalcontrast layer b_(T) ^(G) is replaced with a locally adapted globalcontrast layer b_(P) ^(GA), which is defined as

b _(P) ^(GA) =b _(P) −b _(P) ^(LA)

Then, the tone-mapping function (3.14) is replaced with its locallyadaptive representation

Δb _(P) ^(A)(b _(P))=−ρ_(L) b _(P) ^(LA) −ρ _(a) b _(P) ^(GA) +Δb_(out)  (3.18)

where ρ_(G)=ρ_(global).

Solution for statement 3 above. Additive local contrast compensation(3.17) calculated at the same point (x,y) for a certain r can givedifferent visual contrast results for different observation distances ofthe image b_(new). In a particular case shown in FIG. 4, equation (3.17)is considered as a dependence of b_(P) ^(LA) calculated for a certain rat the point o situated at a distance r′ from a transition edge e:visual luminance distribution along l direction will depend on itsobservation distance. To make details (luminance transitions) of theoutput image visually less dependent on a distance of their observation,an effective locally adaptive tone mapping function (LATM) is built as asuperposition of different responses of b_(P) ^(LA) at (x,y) over allavailable r. This can be done through an integral of (3.18) over rparameter

$\begin{matrix}{{\Delta \; {b_{A}( b_{P} )}} = {\frac{1}{R_{\max}}{\int_{0}^{R_{\max}}{\Delta \; {b_{r}^{A}( b_{P} )}{{dr}.}}}}} & (3.191)\end{matrix}$

The R_(max) sets the maximal available value r for the given image size;it can be a user-defined value in the algorithm applications.

The equation (3.19) can be rewritten as

$\begin{matrix}{{{{\Delta \; {b_{A}( b_{P} )}} = {{{- \rho_{L}}b_{A}^{L}} - {\rho_{G}b_{A}^{G}} + {\Delta \; b_{out}}}}{where}{b_{A}^{L} = {{b_{A}^{L}( {x,y} )} = {{\frac{1}{R_{\max}}{\int_{0}^{R_{\max}}{{b_{r}^{LA}( {x,y} )}{{dr}.b_{A}^{G}}}}} = {{b_{A}^{G}( {x,y} )} = {{b_{P}( {x,y} )} - {b_{A}^{L}( {x,y} )}}}}}}{{b_{r}^{LA}( {x,y} )} = {\frac{h_{s}{{b_{r}^{L}( {x,y} )}}}{1 + {h_{s}{{b_{r}^{L}( {x,y} )}}}}{b_{r}^{L}( {x,y} )}}}{h_{s} = {{\frac{1}{b_{\max}^{L}}.{b_{r}^{L}( {x,y} )}} = {{b_{P}( {x,y} )} - {{\overset{\_}{b}}_{r}( {x,y} )}}}}}\;} & (3.20)\end{matrix}$

The b _(P)(x,y) layer can be calculated from (3.13) or using some othersuitable low-pass filtering.

Parameters Δb_(out), b_(max) ^(L), ρ_(L) and ρ_(G) are user-definedconstants, which don't depend on (x,y,r). Parameters ρ_(L) and ρ_(G) areintended for manual adjustments of local and global contrast correctionsaccordingly, b_(max) ^(L) is a contrast limit for a reproduction deviceΔb_(out) and is a total output exposure offset.

Following is a description of a hardware implementation of the highdynamic range imaging of the present invention in an HDR video device.

For the input data form, the video stream consists of consecutive images(frames), produced by an image sensor or a set of image sensors.

For the given implementation of the HDR algorithms, it is supposed thatthe frames can flow as either a single video stream of consecutiveframes or multiple parallel video streams.

For the proposed HDR processing, which is the number of the frame is notimportant. The only parameter here is an exposure setting

$ɛ = \frac{\alpha \; \tau}{f^{2}}$

where τ is an exposure time, α is a gain, f relates to an aperture.

For input data preparation, let “I” represent an internal m-bit colorpixel data produced by a digital output from a Color Image Sensor (CIS).Value I=I(x,y) will be assumed here as a linear brightnessrepresentation of the incident light power in the pixel, where (x,y) arecoordinates of the pixel in the image. The pixel data are in the rangeof [0,l₀], where l₀ is a maximal possible positive value. For m-bit data

l ₀=2^(M)−1

The merging algorithm is intended for mosaicked RGGB images, so therewill be four kinds of “color” pixels l_(r), l_(g1), l_(g2) and l_(b),depending on the (x, y) position. Since the merging procedure merges theimages pixel-by-pixel regardless of neighboring colors, the brightnessof each pixel will be processed independently. Thus, the input for themerging procedure will be in the form of just l=l(x,y), without thecolor filters notation.

The merging procedure uses a logarithmic (by 2-basis) representation ofthe pixel brightness in terms of exposure values

b _(in)=log₂ l−log₂ l ₀  (4.1)

In (4.1) the B_(in) is “normalized” to be always less than “0” for anyl.

Usually, the transfer function of pixel-to-ADC conversion is not linear,so a preliminary linearization is preferable to be done beforecalculations. Nevertheless, some dominating gamma-like nonlinearity isbeing compensated by the following calculation

b _(corr) =C _(rs)*(b _(in) +b _(ref))−i b_(ref)  (4.2).

The calculation (4.2) has a constant reference value b_(ref) andde-gamma coefficient C_(rs). Both values do not depend on (x,y).

Exposure setting for each frame (so called ‘bracket’) will be definedhere through an EV offset EV_(offs) ^((n)) for each image, where n is anumber of a bracket (exposure setting)

${EV}_{offs}^{(n)} = {\log_{2}\frac{ɛ_{0}}{ɛ_{\pi}}}$

where ε₀ is a <<reference>> exposure. Before the merging procedure, theimage is recalculated in accordance with its appropriate exposuresetting as

b _(LDR)(x,y)=b _(corr)(x,y)+EV _(offs) ^((n))  (4.3)

For the input parameters of the merging algorithm:

-   -   1. b_(hl)—threshold level for highlights in LDR frame.    -   2. EV_(offs) ^((n))—EV offset for the given LDR frame.    -   3. b_(HR)—maximal dynamic range of HDR image in EV values;        b_(HR)>0.    -   4. Q—ghost suppression factor; Q>0    -   5. α_(mix)—mixing parameter; 0≦α_(mix)≦1.

For memory allocations:

-   -   1. b_(LDR)=b_(LDR)(x,y) is a buffer for the input LDR image        obtained in the Input

Data Preparation section.

-   -   2. b_(HDR)=b_(HDR)(x,y) is a buffer for the resulting HDR image        of the same size (in pixels) as the input LDR image.    -   3. G_(kernel) is a Gaussian distributed 2D-kernel (coefficients        of a low-pass Gaussian filter).

The merging algorithm merges pixels of an image b_(LDR)(x,y) with thegiven EV_(offs) ^((n)) into an image b_(HDR)(x,y) by the following way:

The buffer b_(HDR)(x,y) is initialized with the first frame n=0 of thebracketing series.

Then, for other brackets (n>0):

1. Finding Absolute Differences

Δb _(LH)(x,y)=|b _(LDR)(x,y)−b _(HDR)(x,y)|

2. Finding Reference Values

variant 1:

Δb _(ref)(x,y)=Q*log₂(1+2^(−b) ^(LDR) ^((x,y)−b) ^(HR) )

variant 2:

${\Delta \; b_{ref}} = {Q*( \frac{b_{LDR}( {x,y} )}{b_{HR}} )^{2}}$

3. Creating a Merging Mask

${M_{mld}( {x,y} )} = \{ {{\begin{matrix}{1,} \\{0,}\end{matrix}\begin{matrix}{{{if}\mspace{14mu} \Delta \; {b_{LK}( {x,y} )}} \leq {\Delta \; {b_{ref}( {x,y} )}}} \\{{{if}\mspace{14mu} \Delta \; {b_{LH}( {x,y} )}} > {\Delta \; {b_{ref}( {x,y} )}}}\end{matrix}{M_{h\; 1}( {x,y} )}} = \{ {{\begin{matrix}{1,} & {{{if}\mspace{14mu} {b_{LDR}( {x,y} )}} \leq b_{W}} \\{0,} & {{{if}\mspace{14mu} {b_{LDR}( {x,y} )}} > b_{W}}\end{matrix}b_{W}} = {{b_{h\; 1} + {{EV}_{offs}^{(n)}{M_{ag}( {x,y} )}}} = {{M_{mld}( {x,y} )}*{M_{hl}( {x,y} )}}}} } $

4. Making an Effective Merging Mask

Variant 1.% Direct merging with a mixing of pixel's values

M _(eff) =M _(h1)(x,y)

Variant 2.% Merging with a de-ghost operation

M _(eff) =M _(ag)(x,y)

5. Applying a Low-Pass Filtering L(M|G_(kernel)) Onto the WholeEffective Mask to Suppress Color Transition Artifacts in Motion Areas

M _(merge)=α_(mix) *L(M _(eff) |G _(kernel))

6. Updating b_(HDR)(x,y) with the Frame b_(LDR)(x,y)

b _(HDR)(x,y)=M _(merge)(x,y)*b _(LDR)(x,y)+(1−M _(merge)(x,y))*b_(HDR)(x,y)

7. Repeating Operations 1-6 for All Remaining Brackets (Frames).

For tone-mapping, the tone-mapping algorithm is intended forBayer-mosaicked images, represented in a logarithmic scale b_(HDR)(x,y).Minimal colored detail is supposed to occupy 3×3 of RGGB pixels. Inorder to keep the detail's color, the b_(HDR)(x,y) image is separatedinto a <<brightness>> component:

b _(br) =b _(br)(x,y)

and <<color>> component

δb _(col) =δb _(col)(x,y)

so

b _(HDR)(x,y)=b _(br)(x,y)+δb _(col)(x,y)

where col denotes an appropriate pixel's primary color r, g1, g2 or b at(x,y) position.

The first component contains just exposure brightness values of theimage details, the second one—colors of the details. Ideally, theδb_(col) color component and the brightness b_(br) component should notdepend on each other.

After the separation, tone-mapping calculations are being performed onb_(br) only.

To achieve this, the brightness b_(br) is separated into a low details(LD) component b_(br) ^(LD) and a high details (HD) component b_(br)^(HD), so

b _(br)(x,y)=b _(br) ^(LD)(x,y)+b _(br) ^(HD)(x,y).

At each pixel coordinate (x,y) the tone mapping will be performed by anoperation

b _(br) ^(TM) =F _(TM)(b _(br) ^(LD) ,b _(br) ^(HD))

The tone-mapping function F_(TM) at each pixel coordinate (x,y) isdefined as:

F _(TM)(b _(br) ^(LD) ,b _(br) ^(HD))=b _(E) +b _(br) ^(LD) ^(eff)+ρ_(G)(b _(W) −b _(br) ^(LD) ^(eff) )+ρ_(L) b _(br) ^(HD) ^(eff)

where

b _(br) ^(LD) ^(eff) (x,y)=b _(br)(x,y)−b _(br) ^(HD) ^(eff) (x,y)

obtained from a locally adaptive processing

b_(br)^(HDeff)(x, y) = T * (b_(br)(x, y) − b_(br)^(LD)(x, y))$T = \{ {{\begin{matrix}{0,} & {{{if}\mspace{14mu} T_{r}} < 0} \\{T_{r},} & {{{if}\mspace{14mu} 0} \leq T_{r} \leq 1} \\{1,} & {{{if}\mspace{14mu} T_{r}} > 1}\end{matrix}T_{r}} = \frac{C_{slope}*( {{\rho_{G}*( {b_{W} - {b_{br}( {x,y} )} + C_{shift}} )} - b_{E}} )}{b_{monitor}}} $

Then the final tone-mapped color image is obtained as

b _(TM)(x,y)=b _(br) ^(TM)(x,y)+δb _(col)(x,y)

where

δb _(col)(x,y)=b _(HDR)(x,y)−b _(br)(x,y)

Parameters of the functions above have the following meanings:

-   -   b_(W)—user-defined white point of the b_(br) image    -   b_(E)—total exposure brightness of the b^(TM) image    -   b_(monitor)—target dynamic range of a reproduction device    -   ρ_(G) 13 Gamma-like global contrast    -   ρ_(L)—Local contrast modification of the image    -   C_(slope)—sensitivity of locally adaptive processing    -   C_(shift)—dead-zone for locally adaptive processing

Brightness component is calculated from b_(HDR)(x,y) through a Gaussianlow-pass filtering using a Gaussian 2D-kernel G_(br)

b _(br) =L(b _(HDR) |G _(br))

Standard deviation of the 2D-kernel distribution should cover at least3×3 block of RGGB pixels, which are supposed to detect a color of theminimal detail.

Refer to FIG. 7, which is a drawing illustrating a 5×5 kernel example ofa 2D-kernel distribution of RGGB pixels.

The low-details component of the HDR image is calculated from the b_(br)through the following operations:

Input parameters:N—number of integration loopsMemory allocations:

$G_{rs} = \begin{bmatrix}1 & 2 & 1 \\2 & 4 & 2 \\1 & 2 & 1\end{bmatrix}$

—resampling low-pass filtering 2D-kernelP_(av)—memory buffer of the HDR image sizeP_(LD)—memory buffer of the HDR image sizeW_(count)—integration counterP_(av) ^(rs)—memory buffer of the HDR image size

Process:

1. Initializing buffers

P _(av)(x,y)=b _(br)(x,y)

P _(LP)(x,y)=b _(br)(x,y)

W_(count):=1

Then repeating N loops of the following operations:2. Set parametersk_(stop)=2^(n)//F Find a resampling step for the given rescaling level;n—number of the loopx_(align)=k_(setp)−1//Alignment parametery_(align)=k_(setp)−1//Alignment parameter3.Finding low-pass P_(av) image for the given P_(step):

$P_{av}^{F} = {{\frac{1}{16}{L( {P_{av}G_{rs}} )}}//{{Apply}\mspace{14mu} a\mspace{14mu} {low}\text{-}{pass}\mspace{14mu} {{filtering}.}}}$

4. Subsample the filtered image P_(av) ^(F) within available P_(av) ^(F)size

P _(av) ^(sub)(x,40 ,y′)=P _(av) ^(F)(k _(step) *x′+x _(align) ,k_(step) *y′+y _(align))

5. Get low-pass filtered HDR image P_(av) ^(rs) by a rescaling of P_(av)^(sub) back to the HDR image size using any available rescaling method.6. Integrate the resulting P_(av) ^(rs) into effective P_(LD)low-details image

P _(LD)(x,y)=P _(LD)(x,y)+P _(av) ^(rs)(x,y)

7. Incrementing integration counter

W _(count) =W _(count) +1

8. Creating a new P_(av) rescaled image of smaller size within availableP_(av) ^(F) size

P _(av)({tilde over (x)},{tilde over (y)})=P _(av) ^(F)(2{tilde over(x)}+1,2{tilde over (y)}+1)

9. Return to the point 2, if n≠N, otherwise normalize the result

${b_{br}^{LD}( {x,y} )} = \frac{P_{LD}( {x,y} )}{W_{count}}$

Conversion of any output image b_(out) (like b_(HDR) or b_(TM)) from EVvalues back to a linear representation can be processed as

l_(out)=2^(b) ^(out) ^(+b) ^(monitor)

-   -   where b_(monitor) is a dynamic range of a reproduction device,        or bit-width of the linear image representation.

It will be apparent to those skilled in the art that variousmodifications and variations can be made to the present inventionwithout departing from the scope or spirit of the invention. In view ofthe foregoing, it is intended that the present invention covermodifications and variations of this invention provided they fall withinthe scope of the invention and its equivalent.

1. A high dynamic resolution video processing method comprising: passinglight through a Bayer filter; capturing the light by an image capturedevice to create a Bayer-mosaic image; performing merging techniques onthe Bayer-mosaic image; wherein, during merging techniques, multipleframes of different exposures are used per high dynamic range capture;performing tone mapping techniques on the Bayer-mosaic image; convertingresults of the merging techniques and the tone mapping techniques to redgreen blue (RGB) data; and outputting a high dynamic range image fromthe red green blue data.
 2. The high dynamic resolution video processingmethod of claim 1, the merging techniques comprising a full-resetmerging mode.
 3. The high dynamic resolution video processing method ofclaim 2, the full-reset merging mode comprising: merging of Nconsecutive frames 1, 2, 3, . . . N into a first high dynamic rangeimage; merging of a next series of N consecutive frames 1, 2, 3, . . . Ninto a second high dynamic range image; and merging subsequent series ofN consecutive frames 1, 2, 3, . . . N into subsequent high dynamic rangeimages.
 4. The high dynamic resolution video processing method of claim1, the merging techniques comprising a low dynamic range-updated mergingmode comprising updating a previous high dynamic range frame with a newlow dynamic range frame data to obtain a new high dynamic range frame.5. The high dynamic resolution video processing method of claim 1, themerging techniques comprising a low dynamic range-updated merging modecomprising updating a high dynamic range frame with low dynamic rangeframes at a frame rate of the low dynamic range frames.
 6. The highdynamic resolution video processing method of claim 2, the full-resetmerging mode comprising creating a high dynamic range frame once allmultiple frames have been captured.
 7. The high dynamic resolution videoprocessing method of claim 1, wherein no de-mosaicing or color spaceconversions are utilized to perform merging and tone mapping techniques.8. The high dynamic resolution video processing method of claim 1,wherein all high dynamic range processing techniques are performed in alogarithmic scale.
 9. The high dynamic resolution video processingmethod of claim 1, the tone mapping techniques comprising performinglocally-adaptive tone mapping to a brightness range compression.
 10. Ahigh dynamic resolution video processing method comprising: passinglight through a Bayer filter; capturing the light by an image capturedevice to create a Bayer-mosaic image; performing merging techniquescomprising a full-reset merging mode and a low dynamic range-updatedmerging mode on the Bayer-mosaic image; wherein, during mergingtechniques, multiple frames of different exposures are used per highdynamic range capture; the full-reset merging mode comprising: mergingof N consecutive frames 1, 2, 3, . . . N into a first high dynamic rangeimage; merging of a next series of N consecutive frames 1, 2, 3, . . . Ninto a second high dynamic range image; and merging subsequent series ofN consecutive frames 1, 2, 3, . . . N into subsequent high dynamic rangeimages; performing tone mapping techniques on the Bayer-mosaic image;converting results of the merging techniques and the tone mappingtechniques to red green blue (RGB) data; and outputting a high dynamicrange image from the red green blue data.
 11. The high dynamicresolution video processing method of claim 10, the low dynamicrange-updated merging mode comprising updating a previous high dynamicrange frame with a new low dynamic range frame data to obtain a new highdynamic range frame.
 12. The high dynamic resolution video processingmethod of claim 10, the low dynamic range-updated merging modecomprising updating a high dynamic range frame with low dynamic rangeframes at a frame rate of the low dynamic range frames.
 13. The highdynamic resolution video processing method of claim 10, the full-resetmerging mode comprising creating a high dynamic range frame once allmultiple frames have been captured.
 14. The high dynamic resolutionvideo processing method of claim 10, wherein no de-mosaicing or colorspace conversions are utilized to perform merging and tone mappingtechniques.
 15. The high dynamic resolution video processing method ofclaim 10, wherein all high dynamic range processing techniques areperformed in a logarithmic scale.
 16. The high dynamic resolution videoprocessing method of claim 10, the tone mapping techniques comprisingperforming locally-adaptive tone mapping to a brightness rangecompression.
 17. A high dynamic resolution video processing methodcomprising: converting linear primary Bayer mosaic signals directly to alogarithmic scale pixel-wise; wherein each R, G1, G2, or B pixel isconverted to its logarithmic value independently; processing the pixelsto obtain a high dynamic range result; and converting the high dynamicrange result from the logarithmic scale back to the primary linear Bayermosaic.